Comparison of Surface Mineral Alteration by Fire at Two Different Scales¶

Group Members: Hannah Rieder, Randi Neff, and Bridget Hass (mentor, National Ecological Observatory Network, NEON)¶

Portfolio Post Author: Hannah Rieder¶

  • Hannah's GitHub Repository Branch
  • Randi's GitHub Repository Branch
  • Randi's Portfolio Post - see the "Capstone Project: Comparison of Surface Mineral Alteration by Fire at Two Different Scales" section

Introduction:¶

This project seeks to answer two questions:

  1. How can Python be used to compare mineral distribution maps from the Earth Surface Mineral Dust Source Investigation (EMIT) instrument with classified surface reflectance data gathered by the NEON Airborne Observation Platform (AOP)?
  2. What are the differences in mineral distribution and surface reflectance between burned and unburned areas? How does fire alter surface minerals?

Study Area:¶

The area of interest is the Soaproot Saddle (SOAP) NEON field site in the Sierra National Forest in California. The SOAP site covers 1440 acres and has an average elevation of 1,210 m. The full elevation range is 1,044 m-1373 m. The land cover of SOAP is primarily evergreen forest and shrub/scrub. More specifically, the forest is mixed conifer and has steep hills with narrow drainages.

Boundary of the Sierra National Forest in California with a blue dot marker for the Soaproot Saddle NEON field site.

The national forest boundary in the plot above came from this FS National Forests Dataset (US Forest Service Proclaimed Forests) dataset.

Additionally, to answer the second question, two fires covering at least part of the SOAP site will be analyzed:

  1. Creek Fire: started on September 4, 2020 and burned 379,895 acres
  2. Blue Fire: started on June 30, 2021 and burned 277 acres.

These fires were chosen primarily because there is NEON surface reflectance data from before and after these fires. Also, at the SOAP site, these fires occurred the closest to when EMIT started gathering data.

neon_aop_spectral_screek_fire.png This figure shows the southern boundary of the Creek Fire in red overlaid on an RGB plot of spectrometer bidirectional reflectance SOAP site data gathered by the NEON AOP in 2024. This image was created with the NEON AOP Google Earth Engine Viewer.

EMIT & NEON Background:¶

The NASA Earth Surface Mineral Dust Source Investigation (EMIT) instrument is located on the International Space Station (ISS). When the ISS is above predetermined arid dust source regions, EMIT uses imaging spectroscopy to measure mineral composition. To do this, EMIT measures the at-sensor radiance and surface reflectance. The surface reflectance is used to create mineralogical maps (See NASA Earthdata, 2023). EMIT data has 60 meter resolution (Green, 2023).

The NSF National Ecological Observatory Network (NSF NEON) consists of 81 field sites covering 20 ecoclimatic domains in 48 continental U.S. states, plus Alaska, Hawaii, and Puerto Rico. Each field site has various sensors and tools to measure biological, physical, chemical, and ecological characteristics. NEON also operates an Airborne Observation Platform (AOP), which is a set of instruments on a light aircraft that collect high resolution remote sensing data at a low altitude (see NEON Airborne Remote Sensing webpage). One of the datasets AOP collects is surface reflectance at 1 meter resolution (see Spectrometer Orthorectified Surface Bidirectional Reflectance).

Why is this work important?¶

There are valuable benefits of using NEON AOP and EMIT data together because of the differences in resolution and strengths. The NEON AOP gathers data at a 1 m spatial resolution, while EMIT has a 60 m spatial resolution. The details captured by the NEON AOP could help fill in gaps in EMIT's data. Also, the NEON data can be used to verify and enhance EMIT data by applying the broad patterns captured by EMIT on a smaller scale. In terms of strengths - EMIT's surface mineral identification may be less useful for areas with dense vegetation since the vegetation may "block" access to the true surface of the land. Since the NEON AOP is flown at a much lower altitude, it may be able to capture more accurate surface reflectance data from densely vegetated areas.

This work is important for fire management in the Sierra National Forest as well. Recently, fires in this national forest are less frequent than they used to be. This "has resulted in the growth of uncharacteristically heavy surface fuels and dense forests, thereby increasing the likelihood of high-severity, stand-replacing fires" (Handler, 2019). These evolved fires greatly affect the forest's ecosystem. Understanding how fires change surface minerals could aid teams that manage prescribed burns and teams that work to restore an area post-burn. Also, as mentioned above, EMIT could struggle to get true surface mineral identifications with the now denser Sierra National Forest, but hopefully the data from the NEON AOP can add details and fill in gaps.

What other work has been done in this area?¶
  • NEON endmember classification tutorial
  • Brief Introduction to Hyperspectral Unmixing
  • Recent Developments in Endmember Extraction and Spectral Unmixing - a book chapter discussing current procedures for endmember extraction and spectral unmixing, including possible ways to use spatial information when identifying endmembers
  • NEON tutorial about reading/plotting with HDF5 hyperspectral data
  • EMIT Data Tutorial Series Workshops Week 1: Intro to EMIT Mission and Data
  • EMIT Data Tutorial Series Workshops Week 2: Working with EMIT Data - The Basics

Data Overview:¶
Dataset Name Short Description Range, Resolution, and Area Data Citation
EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty 60 m V001 (EMITL2BMIN) Imaging spectroscopy is used to capture mineralogical measurements of sunlit regions of interest between 52° N latitude and 52° S latitude August 09, 2022 - ongoing, 60 m spatial resolution, each granule is abt 75 km by 75 km Green, R. (2023). EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty 60 m V001 [Dataset]. NASA Land Processes Distributed Active Archive Center. https://doi.org/10.5067/EMIT/EMITL2BMIN.001
NEON - Site management and event reporting Records of land management activities, disturbances, and other notable ecological events for all NEON aquatic and terrestrial sites July 2012 - ongoing; data has date, location, type, and numerical extent of event if possible NEON (National Ecological Observatory Network). Site management and event reporting (DP1.10111.001), RELEASE-2025. https://doi.org/10.48443/af3c-5w64. Dataset accessed from https://data.neonscience.org/data-products/DP1.10111.001/RELEASE-2025 on May 2, 2025.
NEON - Spectrometer orthorectified surface directional reflectance - mosaic* Hyperspectral raster distributed in an open HDF5 format in UTM projection June 2013 - ongoing; 1 m spatial resolution; each file contains all 426 reflectance bands for a single 1 km by 1 km tile NEON (National Ecological Observatory Network). Spectrometer orthorectified surface directional reflectance - mosaic (DP3.30006.001), RELEASE-2025. https://doi.org/10.48443/49kq-8q12. Dataset accessed from https://data.neonscience.org/data-products/DP3.30006.001/RELEASE-2025 on April 28, 2025.

| NEON - Flight Boundary Dataset Shapefile | NEON AOP flight boundaries in shapefile polygons | (0-1 unitless, scaled by 10,000) | See the “Flight Boundaries” section on this site: https://www.neonscience.org/data-samples/data/spatial-data-maps | Fire Perimeter Boundary from CalFire | Fire perimeters for fires greater than 50 acres in California between 2019 and 2023 | resolution unknown | California Fire Perimeters last 5 years View—Overview. (n.d.). Retrieved May 2, 2025, from https://www.arcgis.com/home/item.html?id=692135b4e6ff47c8adae066ff477f4f1#overview |

\In the code below, I used the NEON - Spectrometer orthorectified surface directional reflectance - mosaic dataset. In our workflow diagram and on our README in our GitHub repository, we reference the NEON - Spectrometer orthorectified surface bidirectional reflectance - mosaic dataset. Whether we use the directional or bidirectional dataset is still to be finalized.*

Methods:¶

Workflow Diagram.png

A general overview:¶

We will focus on the NEON Soaproot Saddle Site (SOAP) in the Sierra National Forest in California. Specifically, we will focus on two wildfires that happened at the SOAP site in 2020 and 2021 - the Creek and Blue fires. Python will be used to download, wrangle, and clip EMIT mineral distribution maps to the SOAP flight box boundaries and the fire perimeters. Python will also be used to download, wrangle, and clip NEON surface reflectance data to the SOAP flight box boundaries and the fire perimeters. These NEON surface reflectance data will be classified using k-means or endmember extraction. Then, the EMIT mineral distribution maps will be compared to the classified NEON surface reflectance data using Python. The EMIT maps and classified NEON data will also be used to compare surface mineral distribution and reflectance between burned and unburned areas.

As of May 2025, we have done some exploratory work with the datasets:¶

NEON Spectrometer orthorectified surface directional reflectance - mosaic exploratory work included using the h5py package to work with HDF5 files for the SOAP site, including:

  • reading and saving .h5 files into the Jupyter Notebook,
  • looking at the items in the .h5 files with the visititems method,
  • extracting reflectance values and reflectance metadata from .h5 files,
  • exploring reflectance wavelength values, and
  • extracting/cleaning/plotting single reflectance bands from a reflectance dataset.

The 03_hr_neon_hyperspectral_download.ipynb Jupyter Notebook contains the full exploration. Here are some highlights of that code (they are commented out as this is just to highlight the code; view the notebook to run the code):

  • Reading, saving, and looking at the contents of the .h5 file:
In [ ]:
# # see what the h5py package contains
# help(h5py)
In [ ]:
# # learn about h5py.File
# h5py.File?
In [ ]:
# # Read the NEON_D17_SOAP_DP3_298000_4107000_reflectance.h5 file to variable f
# # Note that you may need to update this filepath for your local machine
# # the 'r' indicates that this is in readonly mode
# f = h5py.File('./03_data/NEON_D17_SOAP_DP3_298000_4107000_reflectance.h5','r')

# # check file
# f
In [ ]:
# #list_dataset lists the names & locations of datasets in an hdf5 file
# def list_dataset(name,node):
#     if isinstance(node, h5py.Dataset):
#         print(name)

# #.visititems lets us look inside the HDF5 dataset
# f.visititems(list_dataset)
In [ ]:
# # call the ls_dataset fxn with .visititems
# f.visititems(ls_dataset)
  • Extracting/cleaning/plotting a single reflectance band:
In [ ]:
# # save a single band of the SOAP reflectance array
# b53 = soap_reflarray[:,:,52].astype(float)

# # check band variable & view band metadata
# print('b53 type:',type(b53))
# print('b53 shape:',b53.shape)
# print('Band 53 Reflectance:\n',b53)
In [ ]:
# #View scale factor and data ignore value
# scaleFactor = soap_reflarray.attrs['Scale_Factor']
# noDataValue = soap_reflarray.attrs['Data_Ignore_Value']
# print('Scale Factor:',scaleFactor)
# print('Data Ignore Value:',noDataValue)

# # apply scale factor and data ignore value
# b53[b53==int(noDataValue)]=np.nan
# b53 = b53/scaleFactor
# print('Cleaned Band 53 Reflectance:\n',b53)
In [ ]:
# # Create plot of band 53
# soap_53_plot = plt.imshow(b53,extent=soap_ext,cmap='Greys') 
# plt.title('SOAP Band 53 Reflectance');
# plt.xlabel("x")
# plt.ylabel("y")

washed out plot of SOAP Band 53 Reflectance

This is a grey scale plot of Band 53 of the NEON SOAP orthorectified surface directional reflectance captured by the NEON AOP in June 2019.

This plot is quite washed out, so let's check the range and distribution of reflectance values with a histogram:

In [ ]:
# # create histogram of band reflectance values to see range & distribution
# plt.hist(b53[~np.isnan(b53)],50); #50 signifies the # of bins
# plt.title("SOAP Band 53 Distribution of Reflectance Values")
# plt.xlabel("reflectance value")
# plt.ylabel("frequency")

histogram of SOAP Band 53 reflectance values

This is a histogram of the Band 53 reflectance values from the NEON SOAP orthorectified surface directional reflectance captured by the NEON AOP in June 2019.

As most of the reflectance values are less than 0.2, we will adjust the color limit on the next plot to make the image less washed out:

In [ ]:
# # adjust color limit on plot
# soap_53_plot = plt.imshow(b53,extent=soap_ext,cmap='Greys',clim=(0,0.2)) 
# plt.title('SOAP Band 53 Reflectance');
# plt.xlabel("x")
# plt.ylabel("y")

SOAP band 53 reflectance

This is a grey scale plot of Band 53 of the NEON SOAP orthorectified surface directional reflectance captured by the NEON AOP in June 2019. The color limit of this plot is limited for reflectance values between 0 and 0.2.

EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty 60 m V001 & NEON - Flight Boundary Dataset exploratory work included using the earthaccess Python library to find EMIT L2B data that intersects with the NEON AOP flight boundaries for the SOAP site.

The 02_hr_finding_co_located_neon_emit_data_soap.ipynb Jupyter Notebook contains the full exploration. Here are some highlights of that code (they are commented out as this is just to highlight the code; view the notebook to run the code):

  • Download flight box data & isolate SOAP flight boundaries:
In [ ]:
# # function to download data stored on the internet in a public url to a local file
# def download_url(url,download_dir):
#     if not os.path.isdir(download_dir):
#         os.makedirs(download_dir)
#     filename = url.split('/')[-1]
#     r = requests.get(url, allow_redirects=True)
#     file_object = open(os.path.join(download_dir,filename),'wb')
#     file_object.write(r.content)
In [3]:
# # Download and Unzip the NEON Flight Boundary Shapefile
# neon_boundary_url = "https://www.neonscience.org/sites/default/files/AOP_flightBoxes_0.zip"
# # Use download_url function to save the file to a directory
# os.makedirs('./02_data', exist_ok=True)
# download_url(neon_boundary_url,'./02_data')
# # Unzip the file
# with ZipFile(f"./02_data/{neon_boundary_url.split('/')[-1]}", 'r') as zip_ref:
#     zip_ref.extractall('./02_data')
    
# # read shapefile of flight boxes to a GeoDataFrame
# aop_flightboxes = gpd.read_file("./02_data/AOP_flightBoxes/AOP_flightboxesAllSites.shp")

# # isolate SOAP boundaries
# site_id = 'SOAP'
# # write this to a new variable called "site_polygon"
# site_polygon = aop_flightboxes[aop_flightboxes.siteID == site_id]

# # make a GeoDataFrame of the SOAP flight boundaries
# site_df = pd.DataFrame({"Name":["SOAP ROI Bounding Box"]})
# site_bbox = gpd.GeoDataFrame({"Name":["SOAP ROI Bounding Box"], "geometry":[site_roi_poly]},crs="EPSG:4326")
# site_bbox
  • Use the convert_bounds function to reformat bounding box coordinates to work with leaflet notation:
In [ ]:
# # Function to convert a bounding box for use in leaflet notation
# def convert_bounds(bbox, invert_y=False):
#     """
#     Helper method for changing bounding box representation to leaflet notation

#     ``(lon1, lat1, lon2, lat2) -> ((lat1, lon1), (lat2, lon2))``
#     """
#     x1, y1, x2, y2 = bbox
#     if invert_y:
#         y1, y2 = y2, y1
#     return ((y1, x1), (y2, x2))
  • Plot SOAP flight boxes:
In [4]:
# fig = Figure(width="750px", height="375px")
# map1 = folium.Map(tiles='https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}', attr='Google')
# fig.add_child(map1)

# # Add Site Bounding Box
# folium.GeoJson(site_bbox, name='bounding_box').add_to(map1)

# # Add site roi geodataframe
# site_polygon.explore("flightbxID",
#                      popup=True,
#                      categorical=True,
#                      cmap='Set3',
#                      style_kwds=dict(opacity=0.7, fillOpacity=0.4),
#                      name="SOAP ROI",
#                      m=map1)

# # write title
# title_html = '''<h3 align="center" style="font-size:16px"><b>NEON SOAP Flight Box Boundaries</b></h3>'''

# map1.add_child(folium.LayerControl())
# map1.fit_bounds(bounds=convert_bounds(site_polygon.unary_union.bounds))

# # add title to map
# map1.get_root().html.add_child(folium.Element(title_html))
# display(fig)

NEON SOAP flight Box boundaries

The background tiles of this plot & the plot below are from Google.

This plot shows the NEON SOAP flight box boundaries used by the NEON AOP.

*To find the EMIT L2B data, first create two functions to get shapely objects for the GeoDataFrames and get certain browse image URLs:

In [ ]:
# # Function to create shapely polygon of spatial coverage
# def get_shapely_object(result:earthaccess.results.DataGranule):
#     # Get Geometry Keys
#     geo = result['umm']['SpatialExtent']['HorizontalSpatialDomain']['Geometry']
#     keys = geo.keys()

#     if 'BoundingRectangles' in keys:
#         bounding_rectangle = geo['BoundingRectangles'][0]
#         # Create bbox tuple
#         bbox_coords = (bounding_rectangle['WestBoundingCoordinate'],bounding_rectangle['SouthBoundingCoordinate'],
#                     bounding_rectangle['EastBoundingCoordinate'],bounding_rectangle['NorthBoundingCoordinate'])
#         # Create shapely geometry from bbox
#         shape = geometry.box(*bbox_coords, ccw=True)
#     elif 'GPolygons' in keys:
#         points = geo['GPolygons'][0]['Boundary']['Points']
#         # Create shapely geometry from polygons
#         shape = geometry.Polygon([[p['Longitude'],p['Latitude']] for p in points])
#     else:
#          raise ValueError('Provided result does not contain bounding boxes/polygons or is incompatible.')
#     return(shape)

# # Retrieve png browse image if it exists or first jpg in list of urls
# def get_png(result:earthaccess.results.DataGranule):
#     https_links = [link for link in result.dataviz_links() if 'https' in link]
#     if len(https_links) == 1:
#         browse = https_links[0]
#     elif len(https_links) == 0:
#         browse = 'no browse image'
#         warnings.warn(f"There is no browse imagery for {result['umm']['GranuleUR']}.")
#     else:
#         browse = [png for png in https_links if '.png' in png][0]
#     return(browse)
  • Now use earthaccess to search for the EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty 60 m V001 dataset using the concept_id (C2408034484-LPCLOUD) and date_range ('2022-01-01','2024-11-01'). Also, create a DataFrame of the query results.
In [ ]:
# # submit a new query for the L2B data
# emit_l2b_query_results = earthaccess.search_data(
#     concept_id=emit_l2b_concept_id,
#     polygon=site_roi,
#     temporal=date_range,
#     count=500)

# # Create Dataframe of Results Metadata
# emit_l2b_results_df = pd.json_normalize(emit_l2b_query_results)
# # Create shapely polygons for result
# geometries = [get_shapely_object(emit_l2b_query_results[index]) for index in emit_l2b_results_df.index.to_list()]
# # Convert to GeoDataframe
# emit_l2b_gdf = gpd.GeoDataFrame(emit_l2b_results_df, geometry=geometries, crs="EPSG:4326")
# # Remove emit_results_df, no longer needed
# del emit_l2b_results_df
# # Add browse imagery links
# emit_l2b_gdf['browse'] = [get_png(granule) for granule in emit_l2b_query_results]
# emit_l2b_gdf['shortname'] = [result['umm']['CollectionReference']['ShortName'] for result in emit_l2b_query_results]
# # Preview GeoDataframe
# print(f'{emit_l2b_gdf.shape[0]} granules total')
  • Visualize intersecting EMIT L2B granules and NEOAN SOAP flight boundaries:
In [ ]:
# # Plot Using Folium
# # Create Figure and Select Background Tiles
# fig = Figure(width="750px", height="375px")
# map1 = folium.Map(tiles='https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}', attr='Google')
# fig.add_child(map1)

# # Add Site Bounding Box
# folium.GeoJson(site_bbox,
#                 name='bounding_box',).add_to(map1)

# # Add roi geodataframe
# site_polygon.explore("flightbxID",
#                       popup=True,
#                       categorical=True,
#                       cmap='Set3',
#                       style_kwds=dict(opacity=0.7, fillOpacity=0.4),
#                       name="SOAP ROI",
#                       m=map1)

# # Plot EMITL2BMIN Results - note we must drop the datetime_obj columns for this to work
# emit_l2b_gdf.drop(columns=['datetime_obj']).explore(
#     "granule",
#     categorical=True,
#     tooltip=[
#         "granule",
#         "start_datetime",
#         "cloud_cover",
#     ],
#     popup=True,
#     style_kwds=dict(fillOpacity=0.1, width=2),
#     name="EMIT",
#     m=map1,
#     legend=False
# )

# # write title
# title_html = '''<h3 align="center" style="font-size:16px"><b>Intersecting Data: EMIT L2B Granules & NEON SOAP Flight Boxes</b></h3>'''

# map1.fit_bounds(bounds=convert_bounds(emit_gdf.unary_union.bounds))
# map1.add_child(folium.LayerControl())

# # add title to map
# map1.get_root().html.add_child(folium.Element(title_html))
# display(fig)

Intersecting EMIT L2B Mineral Granules &amp; NEON SOAP Flight Boxes

These are about 20 different EMIT L2B Mineral dataset granules overlaid on the NEON SOAP Flight Box boundaries. It is a bit hard to see, but the EMIT granules are all of the colorful diamond shapes and the NEON SOAP flight boxes are the squares and rectangles in the middle.

Conclusion/Summary:¶

As we are still in the exploratory phase of this project, there are not many conclusions to draw. We can say that at the NEON SOAP site, there is overlapping data between the EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty dataset and the NEON - Flight Boundary Dataset dataset. This bodes well for being able to find NEON Spectrometer Orthorectified Surface Directional Reflectance data and EMIT L2B Mineral Identification data at the SOAP site which can be used to create classified data and mineral distribution maps, respectively. We also know that we can download, inspect, and plot the NEON surface reflectance data.

It is also important to mention the work we did to set up our coding environments for this project. We had to troubleshoot several issues with installing mamba, creating the environment for the repository, and installing packages for the repository. We have made great strides in learning how to use Jupyter Notebooks and Git Bash for our coding application and commits instead of VS Code. We also learned about different ways to set up a repository for a project such as this and have enacted a folder and file naming structure for us to use moving forward.

Regarding next steps, we will finish downloading the data we have yet to access, hopefully with URLs/APIs. We need to clip all data to the fire and NEON site areas of interest and load all data into easy-to-work with arrays, DataFrames, GeoDataFrames, etc. We will create our mineral distribution maps from the EMIT data and classify the NEON reflectance data (including learning more about our clustering options and deciding which one to use). We then need to decide the best way to compare the mineral maps and classified data, and complete that comparison. Additionally, throughout this process, we need to be thinking ahead to ensure our notebooks are reproducible and well commented. This will make it easier for us to synthesize our notebooks into a Python tutorial about scaling EMIT and NEON data for others to use.

Extra sources not directly linked in text above:¶
  1. DP3.30006.001 | Spectrometer orthorectified surface directional reflectance—Mosaic | NSF NEON | Open Data to Understand our Ecosystems. (n.d.). Retrieved April 28, 2025, from https://www.neonscience.org/taxonomy/term/6146
  2. Frequently Asked Questions (FAQ) | NSF NEON | Open Data to Understand our Ecosystems. (n.d.). Retrieved April 6, 2025, from https://www.neonscience.org/about/faq
  3. Green, R. (2023). EMIT L2B Estimated Mineral Identification and Band Depth and Uncertainty 60 m V001 [Dataset]. NASA Land Processes Distributed Active Archive Center. https://doi.org/10.5067/EMIT/EMITL2BMIN.001
  4. Handler, C. (2019). Sierra National Forest Forestwide Prescribed Fire Project: Updated Purpose, Need and Proposed Action. United States Department of Agriculture. https://firerestorationgroup.org/snf-forestwide-rx-burn
  5. NASA Earthdata (Director). (2023, April 18). EMIT Data Tutorial Series Workshops Week 1: Intro to EMIT Mission and Data [Video recording]. https://www.youtube.com/watch?v=XzSEqdiS2aE
  6. Soaproot Saddle NEON | NSF NEON | Open Data to Understand our Ecosystems. (n.d.). Retrieved April 27, 2025, from https://www.neonscience.org/field-sites/soap